Bayesian inference and experimental design for large generalised linear models

Decision making in light of uncertain and incomplete knowledge is one of the central themes in statistics and machine learning. Probabilistic Bayesian models provide a mathematically rigorous framework to formalise the data acquisition process while making explicit all relevant prior knowledge and assumptions. The resulting posterior distribution represents the state of knowledge of the model and serves as the basis for subsequent decisions. Despite its conceptual clarity, Bayesian inference computations take the form of analytically intractable high-dimensional integrals in practise giving rise to a number of randomised and deterministic approximation techniques. This thesis derives, studies and applies deterministic approximate inference and experimental design algorithms with a focus on the class of generalised linear models (GLMs). Special emphasis is given to algorithmic properties such as convexity, numerical stability, and scalability to large numbers of interacting variables. After a review of the relevant background on GLMs, we introduce the most promising approaches to estimation, approximate inference and experiment design. We study in depth a particular approach and reveal its convexity properties naturally leading to a generic and scalable inference algorithm. Furthermore, we are able to precisely characterise the relationship between Bayesian inference and penalised estimation: estimation is a special case of inference and inference can be done by a sequence of smoothed estimation steps. We then compare a large body of inference algorithms on the task of probabilistic binary classification using a kernelised GLM: the Gaussian process model. Multiple empirical comparisons identify expectation propagation (EP) as the most accurate algorithm. As a next step, we apply EP to adaptively and sequentially design the measurement architecture for the acquisition of natural images in the context of compressive sensing (CS), where redundancy in signals is exploited to accelerate the measurement process. We observe in comparative experiments differences between adaptive CS results in practise and the setting studied in theory. Combining the insights from adaptive CS with our convex variational inference algorithm, we are able – by sequentially optimising Bayesian design scores – to improve the measurement sequence in magnetic resonance imaging (MRI). In our MRI application on realistic image sizes, we achieve scan time reductions for constant image quality.